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Abstract 

We  present  Minimum  Bayes-Risk  (MBR)  de¬ 
coding  for  statistical  machine  translation.  This 
statistical  approach  aims  to  minimize  expected 
loss  of  translation  errors  under  loss  functions 
that  measure  translation  performance.  We  de¬ 
scribe  a  hierarchy  of  loss  functions  that  incor¬ 
porate  different  levels  of  linguistic  information 
from  word  strings,  word-to-word  alignments 
from  an  MT  system,  and  syntactic  structure 
from  parse-trees  of  source  and  target  language 
sentences.  We  report  the  performance  of  the 
MBR  decoders  on  a  Chinese-to-English  trans¬ 
lation  task.  Our  results  show  that  MBR  decod¬ 
ing  can  be  used  to  tune  statistical  MT  perfor¬ 
mance  for  specific  loss  functions. 

1  Introduction 

Statistical  Machine  Translation  systems  have  achieved 
considerable  progress  in  recent  years  as  seen  from  their 
performance  on  international  competitions  in  standard 
evaluation  tasks  (NIST,  2003).  This  rapid  progress  has 
been  greatly  facilitated  by  the  development  of  automatic 
translation  evaluation  metrics  such  as  BLEU  score  (Pa- 
pineni  et  al.,  2001),  NIST  score  (Doddington,  2002) 
and  Position  Independent  Word  Error  Rate  (PER)  (Och, 
2002).  However,  given  the  many  factors  that  influence 
translation  quality,  it  is  unlikely  that  we  will  find  a  single 
translation  metric  that  will  be  able  to  judge  all  these  fac¬ 
tors.  Eor  example,  the  BLEU,  NIST  and  the  PER  metrics, 
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though  effective,  do  not  take  into  account  explicit  syntac¬ 
tic  information  when  measuring  translation  quality. 

Given  that  different  Machine  Translation  (MT)  eval¬ 
uation  metrics  are  useful  for  capturing  different  aspects 
of  translation  quality,  it  becomes  desirable  to  create  MT 
systems  tuned  with  respect  to  each  individual  criterion.  In 
contrast,  the  maximum  likelihood  techniques  that  under¬ 
lie  the  decision  processes  of  most  current  MT  systems  do 
not  take  into  account  these  application  specific  goals.  We 
apply  the  Minimum  Bayes-Risk  (MBR)  techniques  devel¬ 
oped  for  automatic  speech  recognition  (Goel  and  Byrne, 
2000)  and  bitext  word  alignment  for  statistical  MT  (Ku¬ 
mar  and  Byrne,  2002),  to  the  problem  of  building  au¬ 
tomatic  MT  systems  tuned  for  specific  metrics.  This  is 
a  framework  that  can  be  used  with  statistical  models  of 
speech  and  language  to  develop  decision  processes  opti¬ 
mized  for  specific  loss  functions. 

We  will  show  that  MBR  decoding  can  be  applied  to 
machine  translation  in  two  scenarios.  Given  an  automatic 
MT  metric,  we  design  a  loss  function  based  on  the  met¬ 
ric  and  use  MBR  decoding  to  tune  MT  performance  un¬ 
der  the  metric.  We  also  show  how  MBR  decoding  can 
be  used  to  incorporate  syntactic  structure  into  a  statistical 
MT  system  by  building  specialized  loss  functions.  These 
loss  functions  can  use  information  from  word  strings, 
word-to-word  alignments  and  parse-trees  of  the  source 
sentence  and  its  translation.  In  particular  we  describe 
the  design  of  a  Bilingual  Tree  Loss  Function  that  can  ex¬ 
plicitly  use  syntactic  structure  for  measuring  translation 
quality.  MBR  decoding  under  this  loss  function  allows 
us  to  integrate  syntactic  knowledge  into  a  statistical  MT 
system  without  building  detailed  models  of  linguistic  fea¬ 
tures,  and  retraining  the  system  from  scratch. 

We  first  present  a  hierarchy  of  loss  functions  for  trans¬ 
lation  based  on  different  levels  of  lexical  and  syntactic 
information  from  source  and  target  language  sentences. 
This  hierarchy  includes  the  loss  functions  useful  in  both 
situations  where  we  intend  to  apply  MBR  decoding.  We 
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then  present  the  MBR  framework  for  statistical  machine 
translation  under  the  various  translation  loss  functions. 
We  finally  report  the  performance  of  MBR  decoders  op¬ 
timized  for  each  loss  function. 

2  Translation  Loss  Functions 

We  now  introduce  translation  loss  functions  to  measure 
the  quality  of  automatically  generated  translations.  Sup¬ 
pose  we  have  a  sentence  F  in  a  source  language  for 
which  we  have  generated  an  automatic  translation  E' 
with  word-to-word  alignment  A'  relative  to  F.  The  word- 
to-word  alignment  A'  specifies  the  words  in  the  source 
sentence  F  that  are  aligned  to  each  word  in  the  transla¬ 
tion  E' .  We  wish  to  compare  this  automatic  translation 
with  a  reference  translation  E  with  word-to-word  align¬ 
ment  A  relative  to  F. 

We  will  now  present  a  three-tier  hierarchy  of  trans¬ 
lation  loss  functions  of  the  form  L{{E',  A'),  {E,  A);F) 
that  measure  {E',A')  against  {E,A).  These  loss  func¬ 
tions  will  make  use  of  different  levels  of  information  from 
word  strings,  MT  alignments  and  syntactic  structure  from 
parse-trees  of  both  the  source  and  target  strings  as  illus¬ 
trated  in  the  following  table. 


Loss  Function 

Functional  Form 

Lexical 

Target  Language  Parse-Tree 
Bilingual  Parse-Tree 

F(F,F') 

L{Te,Te>) 

L{{Te,A),{Te',A')-Tf) 

We  start  with  an  example  of  two  competing  English 
translations  for  a  Chinese  sentence  (in  Pinyin  without 
tones),  with  their  word-to-word  alignments  in  Figure  1. 
The  reference  translation  for  the  Chinese  sentence  with 
its  word-to-word  alignment  is  shown  in  Figure  2.  In  this 
section,  we  will  show  the  computation  of  different  loss 
functions  for  this  example. 

2.1  Lexical  Loss  Functions 

The  first  class  of  loss  functions  uses  no  informa¬ 
tion  about  word  alignments  or  parse-trees,  so  that 
L{{E',  A'),  {E,  A)-F)  can  be  reduced  to  L{E,  E').  We 
consider  three  loss  functions  in  this  category:  The  BLEU 
score  (Papineni  et  al.,  2001),  word-error  rate,  and  the 
position-independent  word-error  rate  (Och,  2002).  An¬ 
other  example  of  a  loss  function  in  this  class  is  the  MT- 
eval  metric  introduced  in  Melamed  et  al.  (2003).  A  loss 
function  of  this  type  depends  only  on  information  from 
word  strings. 

BLEU  score  (Papineni  et  al.,  2001)  computes  the 
geometric  mean  of  the  precision  of  n-grams  of  vari¬ 
ous  lengths  (n  G  {1..^})  between  a  hypothesis  and 
a  reference  translation,  and  includes  a  brevity  penalty 
(7(F,  E')  <  1)  if  the  hypothesis  is  shorter  than  the  refer¬ 


ence.  We  use  N  =  A. 

BLEU{E,  E’)  =  exp  log  j  E'), 

where  pn{E,E')  is  the  precision  of  n-grams  in  the  hy¬ 
pothesis  E' .  The  BLEU  score  is  zero  if  any  of  the  n-gram 
precisions  Pn{E,E')  is  zero  for  that  sentence  pair.  We 
note  that  0  <  BLEU{E,E')  <  1.  We  derive  a  loss 
function  from  BLEU  score  as 

Fbleu(-®.^')  =  1  -  BLEU{E,E'). 

Word  Error  Rate  (WER)  is  the  ratio  of  the  string-edit 
distance  between  the  reference  and  the  hypothesis  word 
strings  to  the  number  of  words  in  the  reference.  String- 
edit  distance  is  measured  as  the  minimum  number  of  edit 
operations  needed  to  transform  a  word  string  to  the  other 
word  string. 

Position-independent  Word  Error  Rate  (PER)  mea¬ 
sures  the  minimum  number  of  edit  operations  needed 
to  transform  a  word  string  to  any  permutation  of  the 
other  word  string.  The  PER  score  (Och,  2002)  is  then 
computed  as  a  ratio  of  this  distance  to  the  number  of 
words  in  the  reference  word  string. 

2.2  Target  Language  Parse-Tree  Loss  Functions 

The  second  class  of  translation  loss  functions  uses  infor¬ 
mation  only  from  the  parse-trees  of  the  two  translations, 
so  that  I/((F,  A),  (F',  A');  F)  =  F(Te,Te/).  This  loss 
function  has  no  access  to  any  information  from  the  source 
sentence  or  the  word  alignments. 

Examples  of  such  loss  functions  are  tree-edit  distances 
between  parse-trees,  string-edit  distances  between  event 
representation  of  parse-trees  (Tang  et  al.,  2002),  and  tree- 
kernels  (Collins  and  Duffy,  2002).  The  computation  of 
tree-edit  distance  involves  an  unconstrained  alignment  of 
the  two  English  parse-trees.  We  can  simplify  this  prob¬ 
lem  once  we  have  a  third  parse  tree  (for  the  Chinese  sen¬ 
tence)  with  node-to-node  alignment  relative  to  the  two 
English  trees.  We  will  introduce  such  a  loss  function  in 
the  next  section.  We  did  not  perform  experiments  involv¬ 
ing  this  class  of  loss  functions,  but  mention  them  for  com¬ 
pleteness  in  the  hierarchy  of  loss  functions. 

2.3  Bilingual  Parse-Tree  Loss  Functions 

The  third  class  of  loss  functions  uses  information  from 
word  strings,  alignments  and  parse-trees  in  both  lan¬ 
guages,  and  can  be  described  by 

F((F,  A),  (F',  A');  F)  =  L{{Te,  A),  (T^, ,  A');  Tp). 

We  will  now  describe  one  such  loss  function  using  the 
example  in  Figures  1  and  2.  Figure  3  shows  a  tree- 
to-tree  mapping  between  the  source  (Chinese)  parse-tree 
and  parse-trees  of  its  reference  translation  and  two  com¬ 
peting  hypothesis  (English)  translations. 


Figure  1:  Two  competing  English  translations  for  a  Chinese  sentence  with  their  word-to-word  alignments. 
E  export  of  high-tech  pro.ducts  in  guangdpng  in  f^t  two  mopthsjjiis  year  reached  3.76  billion  US  dollars 


jin-nian  qian  liangyue  guangdong  gao  xinjishu  chanpin  chukou  sanqidianliuyi  meiyuan 


Figure  2:  The  reference  translation  for  the  Chinese  sentence  from  Figure  1  with  its  word-to-word  alignments.  Words  in 
the  Chinese  (English)  sentence  shown  as  unaligned  are  aligned  to  the  NULL  word  in  the  English  (Chinese)  sentence. 


We  hrst  assume  that  a  node  n  in  the  source  tree  Tp  can 
be  mapped  to  a  node  m  in  T  (and  a  node  m'  in  T')  using 
word  alignment  A  (and  A'  respectively).  We  denote  the 
subtree  of  T  rooted  at  node  m  by  tm  and  the  subtree  of 
T'  rooted  at  node  m'  by  We  will  now  describe  a 
simple  procedure  that  makes  use  of  the  word  alignment 
A  to  construct  node-to-node  alignment  between  nodes  in 
the  source  tree  Tp  and  the  target  tree  T. 

2.3.1  Alignment  of  Parse-Trees 

For  each  node  n  in  the  source  tree  Tp  we  consider  the 
subtree  rooted  at  n.  We  hrst  read  off  the  source  word 
sequence  corresponding  to  the  leaves  of  We  next  con¬ 
sider  the  subset  of  words  in  the  target  sentence  that  are 
aligned  to  any  word  in  this  source  word  sequence,  and 
select  the  leftmost  and  rightmost  words  from  this  sub¬ 
set.  We  locate  the  leaf  nodes  corresponding  to  these  two 
words  in  the  target  parse  tree  T,  and  obtain  their  closest 
common  ancestor  node  m  G  T.  This  procedure  gives  us 
a  mapping  from  a  node  n  £  Tp  to  a  node  m  £  T  and  this 
mapping  associates  one  subtree  G  Tp  to  one  subtree 


2.3.2  Loss  Computation  between  Aligned 
Parse-Trees 

Given  the  subtree  alignment  between  Tp  and  T,  and 
Tp  and  T',  we  hrst  identify  the  subset  of  nodes  in  Tp  for 
which  we  can  identify  a  corresponding  node  in  both  T 
and  T'. 

Np  =  {n  G  Tf  :  m  ^  e  n  m'  7^  e}. 


The  Bilingual  Parse-Tree  (BiTree)  Loss  Function  can 
then  be  computed  as 

BiTreeLoss((T£,  A),  (Tg/,  A');  Tf)  =  ^ 

nENp 

(1) 

where  is  a  distance  measure  between  sub-trees 

t  and  t'.  Specihc  Bi-tree  loss  functions  are  determined 
through  particular  choices  of  d.  In  our  experiments,  we 
used  a  0/1  loss  function  between  sub-trees  t  and  t' . 

=  {  i  LteLse.  ® 

We  note  that  other  tree-to-tree  distance  measures  can 
also  be  used  to  compute  d,  e.g.  the  distance  function 
could  compare  if  the  subtrees  t  and  t'  have  the  same 
headword/non-terminal  tag. 

The  Bitree  loss  function  measures  the  distance  be¬ 
tween  two  trees  in  terms  of  distances  between  their  cor¬ 
responding  subtrees.  In  this  way,  we  replace  the  string- 
to-string  (Levenshtein)  alignments  (for  WER)  or  n-gram 
matches  (for  BLEU/PER)  with  subtree-to-subtree  align¬ 
ments. 

The  Bitree  Error  Rate  (in  %)  is  computed  as  a  ratio  of 
the  Bi-tree  Loss  function  to  the  number  of  nodes  in  the 
set  Np. 

The  complete  node-to-node  alignment  between  the 
parse-tree  of  the  source  (Chinese)  sentence  and  the  parse 
trees  of  its  reference  translation  and  the  two  hypothesis 
translations  (English)  is  given  in  Table  1.  Each  row  in 
this  table  shows  the  alignment  between  a  node  in  the  Chi¬ 
nese  parse-tree  and  nodes  in  the  reference  and  the  two  hy¬ 
pothesis  parse-trees.  The  computation  of  the  Bitree  Loss 
function  and  the  Bitree  Error  Rate  is  presented  in  the  last 
two  rows  of  the  table. 


T  :  Reference  Translation  (English) 
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export  of  NP4  PP2  in  NP7  NP8  3.76  billion  US  dollars 
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Tf  :  Source  Sentence( Chinese) 
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the  first  two  months  of  DT2  NNl  NNPl  high-tech  products  3.76  billion  US  dollars 


this  year  NNPl  POSl  high-tech  products  3.76  billion 


this  year  Guangdong 

Ti  .•  Hypothesis  Translation  1  (English) 


T2  :  Hypothesis  Translation  2  (English) 


Figure  3:  An  example  showing  a  parse-tree  for  a  Chinese  sentence  and  parse-trees  for  its  reference  translation  and 
two  competing  hypothesis  translations.  We  show  a  sample  alignment  for  one  of  the  nodes  in  the  Chinese  tree  with  its 
corresponding  nodes  in  the  three  English  trees.  The  complete  node-to-node  alignment  between  the  parse-trees  of  the 
Chinese  sentence  and  the  three  English  sentences  is  given  in  Table  1 . 


Node  n  G  Tf 

Node  m  G  T 

Node  mi  G  Ti 
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NP5 

0 

NPl 
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BiTree  Loss 

Loss(F,  Fi) 

17 

Loss(F,  E2) 

24 

BiTree  Error  Rate  (%) 

17/26  =  65.4 

24/26  =  92.3 

Table  1 :  Bi-Tree  Loss  Computation  for  the  parse-trees  shown  in  Figure  3.  Each  row  shows  a  mapping  between  a  node 
in  the  parse-tree  of  the  Chinese  sentence  and  the  nodes  in  parse-trees  of  its  reference  translation,  hypothesis  translation 
1  and  hypothesis  translation  2. 


2.4  Comparison  of  Loss  Functions 

In  Table  2  we  compare  various  translation  loss  functions 
for  the  example  from  Figure  1 .  The  two  hypothesis  trans¬ 
lations  are  very  similar  at  the  word  level  and  therefore  the 
BLEU  score,  PER  and  the  WER  are  identical.  However 
we  observe  that  the  sentences  differ  substantially  in  their 
syntactic  structure  (as  seen  from  Parse-Trees  in  Figure  3), 
and  to  a  lesser  extent  in  their  word-to-word  alignments 
(Figure  1)  to  the  source  sentence.  The  first  hypothesis 
translation  is  parsed  as  a  sentence  S  — >•  NP  V P  while 
the  second  translation  is  parsed  as  a  noun  phrase.  The  Bi¬ 
tree  loss  function  which  depends  both  on  the  parse-trees 
and  the  word-to-word  alignments,  is  therefore  very  differ¬ 
ent  for  the  two  translations  (Table  2).  While  string  based 
metrics  such  as  BLEU,  WER  and  PER  are  insensitive  to 
the  syntactic  structure  of  the  translations,  BiTree  Loss  is 
able  to  measure  this  aspect  of  translation  quality,  and  as¬ 
signs  different  scores  to  the  two  translations. 

We  provide  this  example  to  show  how  a  loss  function 
which  makes  use  of  syntactic  structure  from  source  and 
target  parse  trees,  can  capture  properties  of  translations 
that  string  based  loss  functions  are  unable  to  measure. 


Loss  Functions 

L{E,E,) 

L{E,E2) 

BLEU  (%) 

26.4 

26.4 

WER  (%) 

70.6 

70.6 

PER  (%) 

23.5 

23.5 

BiTree  Error  Rate  (%) 

65.4 

92.3 

Table  2:  Comparison  of  the  different  loss  functions  for 
hypothesis  and  reference  translations  from  Figures  1,  2. 

3  Minimum  Bayes-Risk  Decoding 

Statistical  Machine  Translation  (Brown  et  ah,  1990)  can 
be  formulated  as  a  mapping  of  a  word  sequence  F  in  a 
source  language  to  word  sequence  E'  in  the  target  lan¬ 
guage  that  has  a  word-to-word  alignment  A'  relative  to  F. 
Given  the  source  sentence  F,  the  MT  decoder  5(F)  pro¬ 
duces  a  target  word  string  E'  with  word-to-word  align¬ 
ment  A' .  Relative  to  a  reference  translation  E  with  word 
alignment  A,  the  decoder  performance  is  measured  as 
L{{E,  A),5{F)).  Our  goal  is  to  find  the  decoder  that  has 
the  best  performance  over  all  translations.  This  is  mea¬ 
sured  through  Bayes-Risk : 

R{6{F))  =  Ep^e,a,f)[L{{E,A),6{F))]. 


The  expectation  is  taken  under  the  true  distribution 
P{E,  A,  F)  that  describes  translations  of  human  quality. 

Given  a  loss  function  and  a  distribution,  it  is  well 
known  that  the  decision  rule  that  minimizes  the  Bayes- 
Risk  is  given  by  (Bickel  and  Doksum,  1977;  Goel  and 
Byrne,  2000); 

5{F)  =  argmin  V  L{{F,  A),  iF',A')- F)P{E,  A\F). 

E,A 

(3) 

We  shall  refer  to  the  decoder  given  by  this  equation 
as  the  Minimum  Bayes-Risk  (MBR)  decoder.  The  MBR 
decoder  can  be  thought  of  as  selecting  a  consensus  trans¬ 
lation:  For  each  sentence  F,  Equation  3  selects  the  trans¬ 
lation  that  is  closest  on  an  average  to  all  the  likely  trans¬ 
lations  and  alignments.  The  closeness  is  measured  under 
the  loss  function  of  interest. 

This  optimal  decoder  has  the  difficulties  of  search 
(minimization)  and  computing  the  expectation  under  the 
true  distribution.  In  practice,  we  will  consider  the  space 
of  translations  to  be  an  N -best  list  of  translation  alterna¬ 
tives  generated  under  a  baseline  translation  model.  Of 
course,  we  do  not  have  access  to  the  true  distribution 
over  translations.  We  therefore  use  statistical  transla¬ 
tion  models  (Och,  2002)  to  approximate  the  distribution 
P{E,A\F). 

Decoder  Implementation:  The  MBR  decoder  (Equa¬ 
tion  3)  on  the  W-best  List  is  implemented  as 

N 

argmin  ,  Aj),  {Ei,  Ai))P{Ej,  Aj\F) 


does  not  distinguish  between  different  types  of  translation 
errors  and  good  translations  receive  the  same  penalty  as 
poor  translations. 

4  Performance  of  MBR  Decoders 

We  performed  our  experiments  on  the  Large-Data  Track 
of  the  NIST  Chinese-to-English  MT  task  (NIST,  2003). 
The  goal  of  this  task  is  the  translation  of  news  stories 
from  Chinese  to  English.  The  test  set  has  a  total  of  1791 
sentences,  consisting  of  993  sentences  from  the  NIST 
2001  MT-eval  set  and  878  sentences  from  the  NIST  2002 
MT-eval  set.  Each  Chinese  sentence  in  this  set  has  four 
reference  translations. 

4.1  Evaluation  Metrics 

The  performance  of  the  baseline  and  the  MBR  decoders 
under  the  different  loss  functions  was  measured  with  re¬ 
spect  to  the  four  reference  translations  provided  for  the 
test  set.  Eour  evaluation  metrics  were  used.  These 
were  multi-reference  Word  Error  Rate  (mWER)  (Och, 
2002),  multi-reference  Position-independent  word  Error 
Rate  (mPER)  (Och,  2002)  ,  BLEU  and  multi-reference 
BiTree  Error  Rate. 

Among  these  evaluation  metrics,  the  BLEU  score 
directly  takes  into  account  multiple  reference  transla¬ 
tions  (Papineni  et  ak,  2001).  In  case  of  the  other  metrics, 
we  consider  multiple  references  in  the  following  way.  Eor 
each  sentence,  we  compute  the  error  rate  of  the  hypothe¬ 
sis  translation  with  respect  to  the  most  similar  reference 
translation  under  the  corresponding  loss  function. 


and  d{E)  —  {E^,A^).  This  is  a  rescoring  procedure  that 
searches  for  consensus  under  a  given  loss  function.  The 
posterior  probability  of  each  hypothesis  in  the  W-best  list 
is  derived  from  the  joint  probability  assigned  by  the  base¬ 
line  translation  model. 


P{{Ej,Aj)\F) 


P{Ej,Aj,F) 

Ef=iPiEi,A„Fy 


(4) 


The  conventional  Maximum  A  Posteriori  (MAP)  de¬ 
coder  can  be  derived  as  a  special  case  of  the  MBR  de¬ 
coder  by  considering  a  loss  function  that  assigns  a  equal 
cost  (say  1)  to  all  misclassifications.  Under  the  0/1  loss 
function. 


L{{E,A),{E\A'))  =  i^  5 


ifE  =  E'  kA  =  A' 
otherwise. 


(5) 

the  decoder  of  Equation  3  reduces  to  the  MAP  decoder 


%AP(-^)  =  (6) 


This  illustrates  why  we  are  interested  in  MBR  decoders 
based  on  other  loss  functions:  the  MAP  decoder  is  opti¬ 
mal  with  respect  to  a  loss  function  that  is  very  harsh.  It 


4.2  Decoder  Performance 

In  our  experiments,  a  baseline  translation  model  (JHU, 
2003),  trained  on  a  Chinese-English  parallel  cor¬ 
pus  (NIST,  2003)  (170M  English  words  and  157M  Chi¬ 
nese  words),  was  used  to  generate  1000-best  translation 
hypotheses  for  each  Chinese  sentence  in  the  test  set.  The 
1000-best  lists  were  then  rescored  using  the  different 
translation  loss  functions  described  in  Section  2. 

The  English  sentences  in  the  W-best  lists  were  parsed 
using  the  Collins  parser  (Collins,  1999),  and  the  Chinese 
sentences  were  parsed  using  a  Chinese  parser  provided  to 
us  by  D.  Bikel  (Bikel  and  Chiang,  2000).  The  English 
parser  was  trained  on  the  Penn  Treebank  and  the  Chinese 
parser  on  the  Penn  Chinese  treebank. 

Under  each  loss  function,  the  MBR  decoding  was  per¬ 
formed  using  Equation  3.  We  say  we  have  a  matched 
condition  when  the  same  loss  function  is  used  in  both  the 
error  rate  and  the  decoder  design.  The  performance  of 
the  MBR  decoders  on  the  NIST  2001H-2002  test  set  is  re¬ 
ported  in  Table  3.  Eor  all  performance  metrics,  we  show 
the  70%  confidence  interval  with  respect  to  the  MAP 
baseline  computed  using  bootstrap  resampling  (Press  et 
ak,  2002;  Och,  2003).  We  note  that  this  significance  level 


does  meet  the  customary  criteria  for  minimum  signifi¬ 
cance  intervals  of  68.3%  (Press  et  al.,  2002). 

We  observe  in  most  cases  that  the  MBR  decoder  under 
a  loss  function  performs  the  best  under  the  correspond¬ 
ing  error  metric  i.e.  matched  conditions  perform  the  best. 
The  gains  from  MBR  decoding  under  matched  conditions 
are  statistically  significant  in  most  cases.  We  note  that  the 
MAP  decoder  is  not  optimal  in  any  of  the  cases.  In  partic¬ 
ular,  the  translation  performance  under  the  BLEU  metric 
can  be  improved  by  using  MBR  relative  to  MAP  decod¬ 
ing.  This  shows  the  value  of  finding  decoding  procedure 
matched  to  the  performance  criterion  of  interest. 

We  also  notice  some  affinity  among  the  loss  functions. 
The  MBR  decoding  under  the  Bitree  Loss  function  per¬ 
forms  better  under  the  WER  relative  to  the  MAP  decoder, 
but  perform  poorly  under  the  BLEU  metric.  The  MBR 
decoder  under  WER  and  PER  perform  better  than  the 
MAP  decoder  under  all  error  metrics.  The  MBR  decoder 
under  BLEU  loss  function  obtains  a  similar  (or  worse) 
performance  relative  to  MAP  decoder  on  all  metrics  other 
than  BLEU. 

5  Discussion 

We  have  described  the  formulation  of  Minimum  Bayes- 
Risk  decoders  for  machine  translation.  This  is  a  general 
framework  that  allows  us  to  build  special  purpose  de¬ 
coders  from  general  purpose  models.  The  procedure  aims 
at  direct  minimization  of  the  expected  risk  of  translation 
errors  under  a  given  loss  function.  In  this  paper  we  have 
focused  on  two  situations  where  this  framework  could  be 
applied. 

Given  an  MT  evaluation  metric  of  interest  such  as 
BLEU,  PER  or  WER,  we  can  use  this  metric  as  a  loss 
function  within  the  MBR  framework  to  design  decoders 
optimized  for  the  evaluation  criterion.  In  particular,  the 
MBR  decoding  under  the  BLEU  loss  function  can  yield 
further  improvements  on  top  of  MAP  decoding. 

Suppose  we  are  interested  in  improving  syntactic  struc¬ 
ture  of  automatic  translations  and  would  like  to  use  an 
existing  statistical  MT  system  that  is  trained  without  any 
linguistic  features.  We  have  shown  in  such  a  situation 
how  MBR  decoding  can  be  applied  to  the  MT  system. 
This  can  be  done  by  the  design  of  translation  loss  func¬ 
tions  from  varied  linguistic  analyzes.  We  have  shown  the 
construction  of  a  Bitree  loss  function  to  compare  parse- 
trees  of  any  two  translations  using  alignments  with  re¬ 
spect  to  a  parse-tree  for  the  source  sentence.  The  loss 
function  therefore  avoids  the  problem  of  unconstrained 
tree-to-tree  alignment.  Using  an  example,  we  have  shown 
that  this  loss  function  can  measure  qualities  of  transla¬ 
tion  that  string  (and  ngram)  based  metrics  cannot  cap¬ 
ture.  The  MBR  decoder  under  this  loss  function  gives 
improvements  under  an  evaluation  metric  based  on  the 
loss  function. 


We  present  results  under  the  Bitree  loss  function  as 
an  example  of  incorporating  linguistic  information  into 
a  loss  function;  we  have  not  yet  measured  its  correla¬ 
tion  with  human  assessments  of  translation  quality.  This 
loss  function  allows  us  to  integrate  syntactic  structure  into 
the  statistical  MT  framework  without  building  detailed 
models  of  syntactic  features  and  retraining  models  from 
scratch.  However,  we  emphasize  that  the  MBR  tech¬ 
niques  do  not  preclude  the  construction  of  complex  mod¬ 
els  of  syntactic  structure.  Translation  models  that  have 
been  trained  with  linguistic  features  could  still  benefit  by 
the  application  of  MBR  decoding  procedures. 

That  machine  translation  evaluation  continues  to  be 
an  active  area  of  research  is  evident  from  recent  work¬ 
shops  (AMTA,  2003).  We  expect  new  automatic  MT 
evaluation  metrics  to  emerge  frequently  in  the  future. 
Given  any  translation  metric,  the  MBR  decoding  frame¬ 
work  will  allow  us  to  optimize  existing  MT  systems  for 
the  new  criterion.  This  is  intended  to  compensate  for  any 
mismatch  between  decoding  strategy  of  MT  systems  and 
their  evaluation  criteria.  While  we  have  focused  on  de¬ 
veloping  MBR  procedures  for  loss  functions  that  mea¬ 
sure  various  aspects  of  translation  quality,  this  frame¬ 
work  can  also  be  used  with  loss  functions  which  measure 
application-specific  error  criteria. 

We  now  describe  related  training  and  search  proce¬ 
dures  for  NLP  that  explicitly  take  into  consideration  task- 
specific  performance  metrics.  Och  (2003)  developed  a 
training  procedure  that  incorporates  various  MT  evalua¬ 
tion  criteria  in  the  training  procedure  of  log-linear  MT 
models.  Eoster  et  al.  (2002)  developed  a  text-prediction 
system  for  translators  that  maximizes  expected  benefit  to 
the  translator  under  a  statistical  user  model.  In  parsing, 
Goodman  (1996)  developed  parsing  algorithms  that  are 
appropriate  for  specific  parsing  metrics.  There  has  also 
been  recent  work  that  combines  1-best  hypotheses  from 
multiple  translation  systems  (Bangalore  et  al.,  2002);  this 
approach  uses  string-edit  distance  to  align  the  hypotheses 
and  rescores  the  resulting  lattice  with  a  language  model. 

In  future  work  we  plan  to  extend  the  search  space  of 
MBR  decoders  to  translation  lattices  produced  by  the 
baseline  system.  Translation  lattices  (Ueffing  et  al.,  2002; 
Kumar  and  Byrne,  2003)  are  a  compact  representation  of 
a  large  set  of  most  likely  translations  generated  by  an  MT 
system.  While  an  W-best  list  contains  only  a  limited  re¬ 
ordering  of  hypotheses,  a  translation  lattice  will  contain 
hypotheses  with  a  vastly  greater  number  of  re-orderings. 
We  are  developing  efficient  lattice  search  procedures  for 
MBR  decoders.  By  extending  the  search  space  of  the  de¬ 
coder  to  a  much  larger  space  than  the  W-best  list,  we  ex¬ 
pect  further  performance  improvements. 

MBR  is  a  promising  modeling  framework  for  statisti¬ 
cal  machine  translation.  It  is  a  simple  model  rescoring 
framework  that  improves  well-trained  statistical  models 


Performance  Metrics  | 

Decoder 

BLEU  (%) 

mWER(%) 

mPER  (%) 

mBiTree  Error  Rate(%) 

70%  Confidence  Intervals 

+/-0.3 

+/-0.9 

+/-0.6 

+/-1.0 

MAP(baseline) 

31.2 

64.9 

41.3 

69.0 

MBR 

1 

BLEU 

31.5 

65.1 

41.1 

68.9 

WER 

31.3 

64.3 

40.8 

68.5 

PER 

31.3 

64.6 

40.4 

68.6 

BiTree  Loss 

30.7 

64.1 

41.1 

68.0 

Table  3:  Translation  performance  of  the  MBR  decoder  under  various  loss  functions  on  the  NIST  2001+2002  Test  set. 
For  each  metric,  the  performance  under  a  matched  condition  is  shown  in  bold.  Note  that  better  results  correspond  to 
higher  BLEU  scores  and  to  lower  error  rates. 


by  tuning  them  for  particular  criteria.  These  criteria  could 
come  from  evaluation  metrics  or  from  other  desiderata 
(such  as  syntactic  well-formedness)  that  we  wish  to  see 
in  automatic  translations. 
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